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Ozet 

Amac: Bu arastirmanin amaci, probleme dayali 6renmenin (PDO) etkinligi- 
ni degerlendirmeye yonelik 6grenci ile egitim yonlendiricisi icin gelistirilen bir 
6lg¢me aracinin gecerlik ve gtivenirlik analizlerini yapmaktir. Gere¢ ve Yontem: 
Literatiirde PDO niin klasik egitime gore 6frenciye daha fazla yarar sagladigi 
kabul edilen konular derlenerek 6grenciler icin, ig alt boyutta (bilgi (5), bece- 
ri (7) ve tutum (7) olmak Uzere) 19 madde, 6gretim elemanlart icin ise tek alt 
boyutta ve 14 maddeden olusan bir élcme araci gelistirilmistir. Olcekteki her 
bir madde icin, 1 (kesinlikle katilmiyorum) ile 5 (tiimiyle katillyorum) arasin- 
da Likert tipi bes secenekten birisinin isaretlenmesi istenmistir. Akdeniz Uni- 
versitesi Tip Fakiiltesi donem | 6grencileri (n= 1265) ve PDO de egitim yon- 
lendiricisi olarak gérev alan 6gretim elamanlarindan (n=392) olusan iki fark- 
li calisma grubu arastirma grubunu olusturmustur. Bulgular: Bu dlcegin yap 
gecerligi incelenirken tim alt boyutlarda Karsilastirmal Uyum indeksi (CFI) 
> 0.9 Standardize Ortalama Hatalarin Karekokii (SRMR) 0.05 ve Yaklasik Ha- 
talarin Ortalama Karekékii (RMSEA) 0.07 olarak bulunmustur. Cronbach Alfa 
degeri bilgi boyutunda 0.72, beceride 0.81, tutum boyutunda 0.73, dlcegin 
tiimii iginse 0.86 olarak hesaplanmis ve kabul edilebilir duizeyde bulunmustur. 
Olcegin 6gretim elemani boyutunun yap! gecerligi incelenirken Karsilastirma- 
li Uyum indeksi (CFI) > 0.9 Standardize Ortalama Hatalarin Karekékii (SRMR) 
0.04 ve Yaklasik Hatalarin Ortalama Karekokii (RMSEA) 0.09 olarak bulun- 
mustur. Cronbach Alfa degeri 0.96 olarak hesaplanmis ve kabul edilebilir dii- 
zeyde bulunmustur. Tartisma: Yapilan analizler sonucunda élcegin 6grenci bo- 
yutunun 16, 6gretim elemani boyutunun 14 madde iizerinden gecerli ve gtive- 
nir bir dlgek oldugu saptanmistir. 
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Abstract 

Aim: The purpose of this study was to make validity and reliability analyses of 
a scale developed for both students and facilitators to evaluate PBL efficien- 
cy in medical education. Material and Method: A measuring tool was devel- 
oped for the students and PBL facilitators by compiling evidence based facts 
in literature showing benefits of PBL against classical education. Scale for 
students (SS) composed of 19 items classified in 3 dimensions (knowledge 
(5), skill (7) and attitude (7)) while the scale for the facilitators (SF) had 14 
items in one dimension. As an answer to the main question the participants 
were asked to give a score for each subject statement on a five item Likert- 
type scale between 1 and 5. First year medical students in Akdeniz University 
Faculty of Medicine (n=1265) and their facilitators in PBL sessions (n=392) 
composed of the study group. Results: Validity analysis results of the SS are 
as follows: Confirmatory Fit Index (CFI) >0.9, Standardized Root Mean Square 
Residual (SRMR)=0.05 and Root Mean Square Error of Approximation (RM- 
SEA) = 0.07. Cronbach Alpha values for knowledge, skill and attitude dimen- 
sions were found 0.72, 0.81 and 0.73 respectively. Total reliability score of 
the SS was calculated 0.86. Validity analysis results of the SF are as follows: 
Confirmatory Fit Index (CFI) >0.9, Standardized Root Mean Square Residual 
(SRMR)=0.04 and Root Mean Square Error of Approximation (RMSEA) = 0.09. 
Cronbach Alpha value was found 0.96. Discussion: Consequently, the SS was 
determined to be valid and reliable for 16 items and the SF for 14 items. 
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Introduction 

Problem-Based Learning (PBL) is an education method being 
more widespread since 1969.One of the most important rea- 
sons for that is their benefit [1]. This education method gives 
the student skills of reasoning while solving problems, analyz- 
ing, and synthetizing, reaching information and commenting. 
Thus providing student with ability of developing knowledge 
and skill in the meantime, synthetizing and analyzing knowl- 
edge, and constant self-learning [2-5]. PBL, teaches the student 
to learn efficiently by being active, questioning, investigating, 
wondering, discussing with a little group instead of being a pas- 
sive receiver. PBL teaches the student how and where to use 
the learned knowledge [6-10]. It helps student improve problem 
solving, make counter hypothesis, learning new knowledge to 
support this hypothesis skills. While doing this, it adds useful 
values for medical purposes to students such as scanning re- 
sources, using and evaluating them, group study and commu- 
nication skills [11-14]. With the effect of these benefits, PBL is 
being used more in faculties of medicine each day. 

One of the important components of education program is the 
evaluation. The reaction of the student for the applied methods, 
whether if it made a difference in student’s knowledge, skill or 
attitude, whether if it reached the goals are main questions to 
be answered in evaluation [15]. It is also important to take note 
of comments of students, facilitators and executives which are 
part of the education programme. It is seen that PBL evalua- 
tion studies in litterateur mostly consists of questions about 
whether PBL is making a difference in contentment, knowledge 
and skill. Also the studies are either on students or on facilita- 
tors. In most of these researches, it is seen that researchers 
come to these conclusions from surveys or exam results. To our 
knowledge, there is no such a tool considering views of medical 
students and PBL facilitators together. 

The purpose of this study was to make validity and reliability 
analyses of a measuring tool which has been used for program 
evaluation purposes since 2003 to determine opinions of stu- 
dents and facilitators on PBL. 


Material and Method 

Data Acquiring Tools 

Scale to Evaluate Efficiency of PBL: When literature was ex- 
amined it is noticed that different suggestions were made 
for evaluation. In this study, the stages suggested by, DeVel- 
lis (2003), Tavsancil (2006), Tezbasaran (1997) were followed 
[16-18]. In 2003, in direction of theoretical definition, present 
measuring tools in litterateur were examined and no record of a 
tool used to evaluate efficiency of PBL was found. In this case, 
acknowledged subjects in which PBL has more benefits than 
classic learning are taken into consideration and a measuring 
tool for students consisting of 19 items with 3 sub dimensions 
(Knowledge (5), skill (8), and attitude(6)) and for facilitators 14 
items with one sub dimension was developed. 

These written 3 items were prepared by taking opinions and 
suggestions of experts in field of medicine. Prepared items were 
then examined and edited by measuring and evaluating expert 
for comprehensibility and finally scale was prepared. 

For each item in scale, 5 Likert type options were given between 
1 (I totally do not agree) and 5 (I totally agree) and the person 


answering the survey was asked to pick the most suitable an- 
swer for him/herself. 


Study Group 

For scale development operations, data was collected from 2 
different study groups. First group was Akdeniz University Fac- 
ulty of Medicine 1st semester students and second group was 
PBL directing facilitators in the same faculty. Data from 1161 
people in first group and from 375 people in second group was 
acquired. First study group was used to determine psychometric 
features of the scale for students while second group was used 
to determine psychometric features of the scale for facilitators. 


Acquiring Data 

Taking legal consent, prepared scale was applied to 1st semes- 
ter students at the end of the semester (May-June) since 2003. 
Also same scale was applied to facilitators working with PBL at 
the end of each semester since 2005 once in two years. 

items in each surveys, were asked as follows; “How did PBL help 
students in the following subjects?” for facilitators, and “How 
did PBL help you in the following subjects?” for students. 


Analyzing Data 

The statistical package for the Social Science (SPSS) was used. 
Group differences were analyzed using univariate analysis of 
variance (ANOVA). A principal component analysis was per- 
formed using Kaiser's criterion (Eigenvalue > 1), followed by an 
oblimin rotation. The internal consistency of the overall scale 
and subscales was measured by Cronbact’s alpha coefficient. 
Confirmatory Factor Analysis (CFA) using LISREL [19]. 

The data showed anunivariant normal distribution, because 
the skewness and kurtosis values placed themselves within the 
range -1.0 and +1.0. The skewness values ranged from -.90 to 
+.11 while the kurtosis values ranged from .-97 to +.19. Also, 
correlation values were examined in order to state whether if 
there is a multiple connection problem or not. It is observed that 
values changed between.134 and .734. Since correlation value 
is below .90 there is no multiple connection problem [20]. 

As results of 3 items in survey for students are very low, they are 
subtracted from analysis and the statistical evaluation is made 
using 16 items. 


Results 

Exploratory Factor Analysis (Scale for students) 

First of all, measures of sampling adequacy were conducted on 
the 17-item Scale for students (SS) to see whether it was suit- 
able for factor analysis. Barttett’s test of sphericity indicated 
a chi square value of 4252,24, p < 0.000, while Kaiser-Meyer- 
Olkin measure of sampling adequacy indicates a value of 0.918. 
When a basic scree-test and eigenvalue at > 1.0 criteria were 
used, four factors were generated from the SS. The scree plot 
suggested that three factors should be extracted (Kline, 1994). 
These three factors, which were rotated through the varimax 
procedure, explained 55.04 percent of the variance (Table 2). 
Factor 1 (four items) accounted for 38.2 percent of the vari- 
ance and measured interference with knowledge. Factor 2 
(seven items) accounted for 13.0 percent of the variance and 
measured salience and skills. Factor 3 (five items) accounted 
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for 5.9 percent of the variance and measured overindulgence 
in attitude. 

The reliabilities of the MSLSS dimensions were assessed by 
Cronbach's a coefficient and each dimension’s item-total cor- 
relations. Here acceptable criteria were =.70 for Cronbach's a 
coefficients [21]. (Table 1). 


Table 1. Exploratory Factor Analysis (varimax rotation) of the Scale for students 
Items (N=1161). 


Common 
Variance 


Factor 2 Factor 3 
(Skill) (Attitude) 


Factor 1 
(Knowledge) 


No Item 


1 | would learn subjects -,791 656 
in PBL in shorter 
time with classic 
education 


2 | would learn subjects -,791 653 
in PBL better with 
classic education 


3 | would have better 570 599 
grades in subjects | 
learned with PBL 


4 PBL would affect my 541 536 
motivation better 
than classic educa- 
tion 

5 PBL,improves my ,678 468 
communication skills 


6 PBL improves my 776 ,669 
reason skills 


7 PBL improves my ,799 633 
problem solving skills 


8 PBL improves my 734 470 
decision skills 


9 PBL helps me ap- ,669 ,289 
proach the patient 
as bio psychosocial 
unity 

10 PBL, helps me link 524 572 
basic sciences like 
anatomy, physiology 
with clinical sciences 


11. PBL helps me tell my 521 556 
knowledge to other 
people 

12  PBLhelps me learn 581 497 
in life-time 

13. PBL help me learn by ,602 476 
myself by research 


14 PBLhelps me use ,703 499 
library, internet more 
in order to reach 
knowledge 


15 PBL helps me study ,682 543 
more than usual 


16 Group study used in 566 ,688 
PBL helps me learn 


Cronbach Alpha 0.72 0.81 0.73 0.86 


Confirmatory factor analysis of Scale for students 

The evaluation of model fit was done by using confirmatory 
factor analysis (CFA). In order to perform the CFA, LISREL 8.7 
was used and the model parameters were estimated by using 


Table 2. Confirmatory Factor Analysis, ML: Method: Goog fit indexes 


maximum likelihood [20]. LISREL 8.5 provides a full range of 
goodness-of-fit measures. The three types of overall model fit 
measures useful in CFA can be represented by absolute, incre- 
mental and parsimonious fit. 

In this study, in order to evaluate the absolute fit, X2 (X2: mini- 
mum fit function test), Root Mean Square Error of Approxima- 
tion (RMSEA), goodness of fit index (GF), and standardized root 
mean square residual (SRMR) were used. Adjusted goodness 
of fit index (AGFI), Normed fit index (NFI), Tucker-Lewis index 
(TLI), comparative fit index (CFI), were used as incremental fit 
measures. The results related to models were summarized in 
Table 2. 

When table-2 was examined, the probability levels of all X2 sta- 
tistics were less than 0.01, indicating a rather poor absolute fit 
[22]. X2 value usually gives significant value with large samples. 
For this reason, instead of using X2 value by itself, it is suggest- 
ed to use calculated value to degree of freedom ratio. This ratio 
(X2/df) is wanted to be below 5. When Table 2 was examined, it 
is seen that X2 value (X2=189.14, sd=71, X2/df=2.66, p=.000) 
is significant. In fit indexes GFl, AGFl, NFI, TLI and CFI values 
above .90 means fine fit [23]. RMSEA and SRMR are wanted to 
be below<0.08. When calculated values are examined, it shows 
that there is an acceptable fit. The item-factor loading esti- 
mates, estimated error variances and t values in Table 3. 


Convergent Validity Scale for students 

Item reliability indicates the amount of variance in an item 
due to the underlying construct rather than to error. Either an 
item reliability of at least 0.50, or a significant t value, or both, 
observed for each item, is considered to be evidence of con- 
vergent validity [24]. As seen from Table. 3, all t-values of the 
items were significant and all item reliabilities were greater 
than 0.50, except one items. 

The composite reliability of each construct is one of the prin- 
cipal measures used in assessing the measurement model and 
commonly used higher value for acceptable composite reliabil- 
ity is 0.70. [25] (Table 3). 


Exploratory Factor Analysis (scale for the facilitators) 

First of all, measures of sampling adequacy were conducted 
on the 14-item SF to see whether it was suitable for factor 
analysis. Barttett’s test of sphericity indicated a chi square 
value of 3806,54; p < 0.000, while Kaiser-Meyer-Olkin mea- 
sure of sampling adequacy indicates a value of 0.937. When 
a basic scree-test and eigenvalue at > 1.0 criteria were used, 
four factors were generated from the SF. The scree plot sug- 
gested that three factors should be extracted. (Kline, 1994) 
These three factors, which were rotated through the varimax 
procedure, explained 58.22 percent of the variance (Table 4). 
Factor 1 (eight items) accounted for 58.22 percent of the vari- 
ance and measured interference with. The reliabilities of the 
MSLSS dimensions were assessed by Cronbach's a coefficient 
and each dimension’s item-total correlations. Here acceptable 

criteria were =.70 for Cronbach's a coefficients [21]. 


2. 2 ja . eye 
e me i A OL Confirmatory factor analysis of scale for the facilitators 
189.14 189.14/71 0.0000 097 0.98 0.97 0.98 0.94 0.052 0.074 The evaluation of model fit was done by using confirmatory 
* p<0.01 


factor analysis (CFA). In order to perform the CFA, LISREL 
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Table 3. Confirmatory Factor Analysis: Maximum likelihood estimates 


8.7 was used and the model parameters were estimated by us- 
ing maximum likelihood [20]. LISREL 8.5 provides a full range of 


No No factor tvalues estimated 
loading error goodness-of-fit measures. The three types of overall model fit 
estimates variances ‘ : 
measures useful in CFA can be represented by absolute, incre- 
1 | would learn subjects in PBL in 0.40 8.48 0.85 , , 
; , ; ; mental and parsimonious fit. 
shorter time with classic education 
2. wold idauireubfucte PBL better: 05a tae 096 In this study, in order to evaluate the absolute fit, X2 (X2: mini- 
with classic education mum fit function test), Root Mean Square Error of Approxima- 
3 | would have better grades in sub- -0.61 328 0.63 tion (RMSEA), goodness of fit index (GF), and standardized root 
jects: llearhed.with PBL mean square residual (SRMR) were used. Adjusted goodness 
4 PBL would affect my motivation 0.34 2.78 0.62 of fit index (AGFI), Normed fit index (NFI), Tucker-Lewis index 
better than classic education ; ; . 
(TLI), comparative fit index (CFI), were used as incremental fit 
5 PBLimproves my communication 0.79 3.56 0.38 : . 
skills measures. The results related to models were summarized in 
6 PBL improves my problem solving 0.63 6.49 0.60 Table 5. 
skills When Table 5 was examined, the probability levels of all X2 sta- 
7 PBL improves my decision skills 0.82 23.71 0.33 tistics were less than 0.01, indicating a rather poor absolute fit 
8 PBL helps me approach the patient 0.77 21.76 0.40 [22]. X2 value usually gives significant value with large samples. 
as bio psychosocial unit! A , rl . or 
ia Y For this reason, instead of using X2 value by itself, it is suggest- 
9 PBL,improves my communication 0.63 16.63 0.60 2 ‘ . 
sie ed to use calculated value to degree of freedom ratio. This ratio 
16: “PBL, Adis. ne link baste seiencas: O68 ijee. EA (X2/df) is wanted to be below 5. When Table 2 was examined, it 
like anatomy, physiology with clini- is seen that X2 value (X2=299.65, sd=70, X2/df=4.28, p=.000) 
eaySclerices is significant. In fit indexes GFl, AGFI, NFI, TLI and CFI values 
nm ee helps - tell my knowledge to 0.29 625 0.92 above .90 means fine fit [23]. RMSEA and SRMR are wanted to 
other people ; ; 
be below<0.08. When calculated values are examined, it shows 
12. PBLhelps me learn in life-time 0.74 20.30 0.45 : ; . . 
that there is an acceptable fit. The item-factor loading esti- 
13. PBL helps me learn by myself by 0.71 18.95 0.50 : ; 
researchi mates, estimated error variances and t values in Table 6. 
14 PBL helps me use library, internet 0.51 12.78 0.74 
more in order to reach knowledge 
Table 6: Confirmatory Factor Analysis: Maximum likelihood estimates 
15 PBLhelps me study more thanusual 0.53 0.72 13.15 
No Item Factor tvalues Estimated 
16 Group study used in PBL helps me 0.49 14.78 0.64 loading error 
learn estimates variances 
1 PBL,improves learning motivation 0.77 7.20 0.41 
Table 4. Exploratory Factor Analysis (varimax rotation) of the scale for the 2 Empat sewers Seka 72) a 
facilitators’ Items (n=375). 3 PBL improves problem solving skills 0.84 9.81 0.29 
No Item Factor 1 Common 4 PBL improves decision skills 0.78 7.51 0.40 
Meveniee 5 PBL helps me approach the patient 0.72 5.88 0.48 
1 PBL, improves learning motivation ,778 394 as bio psychosocial unity 
2 PBL, improves reason skills 805 ,637 6 PBL improves communication skills 0.69 4.92 0.52 
3 PBL improves problem solving skills 834 591 7 Improves integrate learned knowl- 0.78 17.62 0.39 
4 PBL improves decision skills 812 421 Bdge Skills 
5 PBL helps me approach the patient as bio psy- ,765 648 S PBL helps pass knowledge:to other 0.76 on 
A ; people 
chosocial unity 
6 PBL improves communication skills 755 695 9 PBL helps life-time learning 0.59 12.09 0.66 
7 Improves integrate learned knowledge skills 792 ,628 10: PBL helps self-learning by research: 0.78 22 O40 
8 PBL helps pass knowledge to other people 769 605 11 Helps use library, internet more in 0.76 17.04 0.42 
order to reach knowledge 
9 PBL helps life-time learnin 649 569 
pee ae 12 Helps filter desired knowledge from 0.69 17.72 0.53 
OQ PBL helps self-learning by research ,798 5644 large subjects 
1 Helps use library, internet more in order to reach ,722 585 13. Helps communications between 0.73 16.22 0.46 
knowledge people and group study 
2 Helps filter desired knowledge from large subjects ,803 521 14 Prepares for Professional life better 0.56 11.38 0.69 
3 Helps communications between people and group 744 ,660 
study 
4 Prepares for Professional life better 628 553 Convergent Validity for scale for the facilitators 
Cronbach Alpha 834 0.96 Item reliability indicates the amount of variance in an item due 
to the underlying construct rather than to error. Either an item 
reliability of at least 0.50, or a significant t value, or both, 
Table 5. Confirmatory Factor Analysis, ML: Method: Goog fit indexes observed for each item, is considered to be evidence of 
2 2 i mr 
x ais i A A convergent validity [24]. As seen from Table. 3, all t-values 
299.65 299.65/70 0.0000 0.97 0.97 0.98 090 085 0.046 0.094 of the items were significant and all item reliabilities were 
“PsQi01 greater than 0.50, except one items. 
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The composite reliability of each construct is one of the prin- 
cipal measures used in assessing the measurement model and 
commonly used higher value for acceptable composite reliabil- 
ity is 0.70 [25] (Table 6). 


Discussion 

As validity and reliability analyze results of scale which is de- 
veloped to evaluate PBL events came out high, both scales are 
considered to useful for evaluation. In student format of the 
scale, 3 items received low values in validity analyses and thus 
these items were subtracted for analyses. 3 subtracted items 
are; “I believe | will be more successful with my Professional life 
in subjects | learned with PBL’, “PBL helps relations between 
people and group study”, and “PBL helped me use internet to 
reach information’. 


Evaluation of Data Acquired from Scale: 

Suggested method to evaluate the scale results; when compar- 
ing arithmetic averages In Likert-type survey items, interval co- 
efficient of classification scale should be calculated using “row 
width/group number” formula. Classification scale was found 
0.80 (5-1=4 and 4/5=0.80). Arithmetic average intervals which 
are taken primarily in evaluating findings are interpreted as fol- 
lows; 1.00-1.80; “I totally do not agree”, 1.81-2.60; “I do not 
agree”, 2.61-3.40; “I partially agree”, 3.41-4.20; “Il agree”, 4.21- 
5.00; “I totally agree’. 


Situations Where PBL Could be used 

All the items in the scale are items in which PBL is specified 
to be more effective than classic learning. For this reason, it is 
thought to be helpful to evaluate the results of the scale whether 
if the results were the same with previous studies. The reasons 
for trying to Show what is already shown before are the fact 
that different infrastructure for each school, different student, 
facilitator profiles, and different applications could give differ- 
ent results. Using the scale, there is a possibility to compare 
different faculties which are using PBL. It is known that PBL 
is becoming widespread in faculties of medicine. While some 
faculties have been using this method for a long time, some 
faculties are new to this subject. Also some faculties structured 
their education programme on PBL; some faculties are using 
hybrid programmes. Each faculty using PBL are expecting to 
benefit advantages and superiorities of this programme. So 
naturally, there is a question about the sufficiency of PBL. It is 
thought that the scale of which validity and reliability analysis 
were completed could be a solution to these questions. 

In both scales, apart from the items which are analyzed, there 
are also suggestions like, “Are you content with PBL?” and “PBL 
is a helpful application for students in general” for facilitators. 
The answers for these suggestions should are expected to be 
“yes”, “ 

validity and reliability analysis. But as it is thought that these 
suggestions could help evaluate the results, they are added to 


no”, or “hesitant”. These suggestions are not included in 


the scale. Also some independent variables are added to fa- 
cilitator scale such as, department of the facilitator (internal 
medicine, surgical etc.), title (prof. Associate prof. Expert etc.) 
how long and how many times the facilitator has been working 
with PBL. The results for these questions are added to instruc- 


tion in order to be help for evaluation. Apart from this, other 
independent variables asked in a different survey could help 
evaluate the result. For example, for students sex, whether if 
they entered faculty of medicine voluntarily, success, and for fa- 
cilitator, sex, age, working period in the faculty questions could 
be helpful for evaluation. 

For a learning method being used more lately in faculties, being 
able to evaluate on a comparable scale in all faculties using the 
method, having both student and facilitator level is a strong 
side of this scale. Also in a large group, having high results in 
validation and reliability analysis is a positive side of this scale. 
Having both student and facilitator level is also a strong side 
of this scale. 


Results 

As a result, as validity and reliability analyze results of scale 
which is developed to evaluate PBL events came out high, both 
student and facilitator scales are considered to be useful in 
faculties which are using PBL for evaluation in acknowledged 
subjects in which PBL is more effective than classic learning. 
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